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ABSTRACT 

In  the  literature  exploring  IS  (information  systems)  evaluation,  practi- 
tioners rarely  practice  what  researchers  preach,  and  researchers  rarely  preach 
what  practitioners  practice.  However,  the  unique  characteristics  of  OA  (office 
automation)  systems  force  a  rethinking  of  the  existing  schism  between  IS 
evaluative  practice  and  research.  This  paper  presents  a  model  of  OA  evalua- 
tion that  integrates  the  perspectives  of  research  and  practice.  The  model  sug- 
gests that  in  certain  circumstances,  purely  quantitative  OA  evaluation  may  be 
irrelevant  or  destructive.  Strategies  for  avoiding  the  undesirable  behavioral 
side-effects  of  OA  evaluations  include  integrating  multiple  perspectives,  con- 
sidering unintended  effects,  using  orthogonal  measures  and  methods,  scruti- 
nizing social  and  decision  processes,  and  discovering  goals. 


INTRODUCTION 

This  paper  is  built  around  a  paradox.  Evaluation  strategies  from  the  in- 
formation systems   (IS)   literature   are   generally   based   upon   quantitative 
methodologies  (e.g.  Schwuchow,  1977;  King  and  Schrems,  1978;  King  and  Ep- 
stein, 1983;  Bozcany,  1983;  Piepta  and  Anderson,  1987;  Sharda,  Barr,  and  Mc- 
Donnell, 1988).  However,  studies  of  IS  professionals  indicate  that  they  fre- 
quently omit  formal,  quantitative  evaluations  of  IS  and  rely  instead  on  more 
intuitive  assessments  of  system  impact  (Greiner,  Leitch,  and  Barnes,  1979; 
Keen,  1981;  Sprague  and  Carlson,  1982;  Couger  and  Colter,  1983;  Hogue  and 
Watson,  1984).  If  formal,  quantitative  models  for  IS  evaluation  are  of  value, 
why  are  they  infrequently  used  by  their  intended  beneficiaries? 

One  explanation  is  that  most  formal  models  of  IS  evaluation  are  ves- 
tiges of  an  era  when  the  goal  of  assessing  IS  was  to  evaluate  the  realized  cost 
savings.  With  the  advent  of  office  automation  (OA)1  however,  IS  increasingly 
impact  qualitative  processes  in  organizations  such  as  social   relationships, 
leadership  emergence,  communication  patterns,  and  managerial  feedback  pro- 
cesses (Williams,  1978;  Zuboff,  1982;  Weick,  1985;  Kiesler,  1986;  Rogers,  1986). 
As  a  result,  the  relevance  of  evaluative  strategies  developed  primarily  to  as- 
sess direct  cost  savings  are  increasingly  suspect.  Further,  given  that  approxi- 
mately 20%  of  U.S.  businesses  have  adopted  some  form  of  office  automation 
(Uttal,  1982;  Rogers  1986)  assessing  the  impact  of  OA  is  far  from  an  "academic" 
issue. 

The  perspective  I  wish  to  consider  in  this  article  is  that  both  the  prac- 
tice and  theory  of  OA  evaluation  can  be  improved  by  a  dialectic  approach. 


OA  is  herein  defined  as  the  application  of  electronic  and  electro-mechanical  devices  to  facilitate 
communications  between  organizational  actors. 


wherein  descriptive  and  normative  views  of  the  evaluative  process  inform  one 
another  (Adler,  1927;  Benson,  1977;  Dirsmith  and  Covaleski,  1987).  To  date, 
approaches  to  IS  evaluation  have  frequently  been  theoretical  constructs  bor- 
rowed from  reference  disciplines  (e.g.  experimental  psychology,  engineering) 
that  ignored  the  context  of  IS,  or,  descriptions  of  existing  IS  practice  un- 
informed by  relevant  theory.  This  article  seeks  to  inform  normative  theory 
with  descriptions  of  practice,  and  descriptions  of  practice  with  normative  the- 
ory. March  argues  for  a  similar  approach  in  discussing  the  relationship  be- 
tween normative  and  descriptive  theories  of  choice: 

"I  think  there  is  good  sense  in  asking  how  the  practical  imple- 
mentation of  theories  of  choice  combine  with  the  ways  people 
behave  when  they  make  decisions,  and  whether  our  ideas  about 
the  engineering  of  choice  might  be  improved  by  greater  attention 
to  our  descriptions  of  choice  behavior."  (March,  1978,  p.  588). 

It  is  worth  noting  that  much  of  the  theory  presented  herein  is  not  new. 
Its  origins  date  to  the  1950s  (e.g.  Blau,  1955;  Merton,  1957).  However,  the  per- 
spective offered  by  this  theory  is  relatively  unexplored  in  IS.  As  a  result,  the 
purpose  of  this  article  is  to  teach  some  old  tricks  (from  social  intervention  and 
organizational  behavior  research)  to  a  new  dog  (OA  evaluation). 

UNDERLYING  PRINCIPLES 

Three  principles  from  organizational  behavior  provide  a  conceptual 
foundation  for  linking  social  intervention  theories  with  OA  evaluation  prac- 
tice. These  are  the  principle  of  incompatibility,  the  distinction  between  high 
and  low  evaluative  methodology,  and  the  distinction  between  uncertainty  and 
ambiguity. 

The  Principle  of  Incompatibility 

The  principle  of  incompatibility  states,  in  essence,  that  as  a  system  be- 
comes more  complex,  it  is  increasingly  difficult  to  generate  statements  about 


the  system  that  are  both  precise  and  significant  (Zadeh,  1973;  Daft  and  Wigin- 
ton,  1979).  Ultimately,  a  threshold  is  reached  beyond  which  precision  and  sig- 
nificance become  mutually  exclusive.  Since  human  systems  are  among  the 
most  complex  of  known  systems  (Boulding,  1956;  Pondy  and  Mitroff,  1978; 
Daft  and  Weick,  1984),  significant  statements  about  organizations  must  nec- 
essarily contain  ambiguity  and  inexactness  to  retain  relevance  (Daft  and  Wig- 
inton,  1979).  Similarly,  assessing  the  impact  of  a  complex  OA  system  necessi- 
tates the  use  of  some  imprecise,  ambiguous  statements  if  evaluation  is  to  be 
nontrivial. 

High  and  Low  Variety  Evaluation  Methodologies 

A  related  distinction  is  the  difference  between  low  and  high  varietv 
evaluation  methodologies  (Pondy,  1977;  Patton,  1987).  Low  variety  evaluation 
methodologies  seek  to  generate  precise  statements  about  organizational  pro- 
cesses, while  sacrificing  accuracy.  Tools  used  in  low  variety  evaluation  include 
traditional  experimental  controls,  formal  statistical  techniques,  and  analytical 
mathematics.  Due  to  the  principle  of  incompatibility,  low  variety  evaluation 
will  likely  detect,  process,  and  exhibit  relatively  less  variety  in  the  system  un- 
der study,  since  they  have  a  limited  capacity  to  detect  qualitative  phenomena 
(Daft  and  Wiginton,  1979).  For  example,  the  use  of  low  variety  methodologies 
might  lead  to  the  conclusion  that  an  implemented  OA  system  had  "no  effect" 
since  there  were  no  measurable  cost  savings  from  the  system. 

In  contrast,  higher  variety  evaluation  methodologies  seek  to  generate 
more  accurate  descriptions,  but  contain  ambiguity  and  imprecision.  Tools  used 
in  higher  variety  evaluation  are  primarily  qualitative  and  are  designed  to  de- 
tect, process,  and  exhibit  qualitative  organizational  phenomena,  such  as  intan- 
gible benefits  and  ultimate  outcomes  (Argyris,  1972;  Child,  1972).  As  a  result 


of  the  use  of  thicker,  richer  measures,  complex  organizational  processes  are 
more  likely  to  be  captured,  understood,  and  explained  using  high  variety 
methodologies  (Weick,  1974). 

Table  1  (adapted  from  Douglas  (1976)  and  Weick  (1984))  provides  a  con- 
tinuum of  evaluation  methodologies.  The  first  three  categories  in  Table  1  rep- 
resent high  variety  evaluation  methodologies.  These  include  instrospection 
(e.g.  a  manager  writing  a  memo)  and  many  forms  of  field  research  (e.g.  a  man- 
ager asking  subordinates  about  system  effects).  The  last  category  in  Table  1 
consists  of  low  variety  evaluation  methodologies,  such  as  questionnaires, 
mathematical  models,  and  statistical  models.  To  date,  evaluative  method- 
ologies proposed  for  evaluating  IS  are  primarily  of  low  variety  (e.g.  King  and 
Schrems,  1978;  Sprague  and  Carlson,   1982;  Sharda,  Barr  and  McDonnell, 
1988),  while  methodologies  used  by  IS  practitioners  are  generally  of  high  vari- 
ety (e.g.  Greiner,  Leitch,  and  Barnes,  1979;  Keen,  1981;  Sprague  and  Carlson, 
1982;  Couger  and  Colter,  1983;  Hogue  and  Watson,  1984). 

Insert  Table  1  about  here 


Uncertainty  and  Equivocality 

A  final  principle  useful  in  considering  OA  evaluative  practices  is  the  dis- 
tinction between  uncertainty  and  equivocality.  Uncertainty  is  the  absence  of 
information  (Miller  and  Frick,  1949;  Daft  and  Lengel,  1986).  As  information  in- 
creases, uncertainty  decreases.  The  game  of  20  questions  illustrates  uncer- 
tainty and  uncertainty  reduction.  A  questioner  receives  yes-no  answers  to 
questions  intended  to  identify  an  unknown  object  as  either  animal,  vegetable, 
or  mineral  (Taylor  and  Faust,  1952;  Bendig,  1953;  Daft  and  Lengel,  1986).  Un- 
certainty is  eliminated  when  the  object  is  correctly  identified.  In  management 


tasks  characterized  by  uncertainty,  managers  are  able  to  ask  questions,  and 
get  answers  that  permit  problem  solving.  Organizational  processes  can  be 
structured  to  reduce  uncertainty  through  the  use  of  rules  and  regulations  and 
through  the  creation  of  formal,  structured  IS  (Daft  and  Lengel,  1986). 

In  contrast,  equivocality  involves  interpreting  data  that  is  unclear,  con- 
flicting, or  paradoxical  (Weick,  1979;  Daft  and  Macintosh,  1981).  The  sentence, 
"I  saw  the  man  on  the  hill  with  the  telescope,"  is  equivocal:  multiple  inter- 
pretations are  possible  (Simon,  1982,  p.  93).  Do  I  have  the  telescope,  or  does 
the  man  on  the  hill?  Is  it  merely  on  the  hill  and  not  in  his  hands?  Managers 
deal  with  'men  on  hills  with  telescopes'  regularly,  and  must  make  sense  of 
such  equivocality.  Daft  et  al.  observe  that  in  equivocal  environments: 

"Managers  are  not  certain  what  questions  to  ask,  and  if  questions 
are  posed  there  is  no  store  of  objective  data  to  provide  an  answer. 
(Daft,  Lengel,  and  Trevino,  1987,  p.  357)." 

Fundamental  to  the  process  of  managing  equivocality  are  exchanges  be- 
tween managers  intended  to  reduce  equivocality  and  create  a  shared  interpre- 
tation that  can  direct  future  events.  When  facing  equivocality,  organizations 
must  enact  solutions  rather  than  discover  them  through  data  gathering  activi- 
ties (Weick,  1979;  Daft  and  Weick,  1984).  The  process  of  enactment  involves 
exchanging  subjective  opinions,  managing  multiple  perspectives,  and  proac- 
tively  shaping  environments  (Smircich  and  Stubbart,  1985).  Equivocality  is  re- 
duced by  managing  and  generating  both  events  and  interpretations  of  events. 

Since  communication  between  managers  is  essential  in  reducing  equiv- 
ocality (Weick,  1979),  implementing  an  OA  system  may  directly  impact  the  or- 
ganizational processes  for  managing  equivocality  and  enacting  solutions  to 
ambiguous  problems  (Daft,  Lengel,  and  Trevino,  1987).  OA  systems  are  fre- 
quently designed  to  alter  communications  patterns  in  tasks  containing  equivo- 


cality.  Neith  decision  support  systems  (DSSs)  designed  for  individual  man- 
agers nor' transaction  processing  systems  are  designed  to  change  organiza- 
tional communication  patterns.  Individualized  DSSs  and  transaction  process- 
ing systems  are  therefore  more  likely  to  impact  organizational  processes  for 
managing  uncertainty,  but  not  equivocality.  For  reasons  to  be  discussed,  the 
use  of  evaluative  methodologies  appropriate  for  uncertainty-managing  sys- 
tems may  lead  to  irrelevant  or  destructive  evaluations  of  equivocality-manag- 
ing systems. 

OA  AS  SOCIAL  EXPERIMENT 

OA  systems  are  social  experiments2,  similar  to  the  implementation  of  a 
program  designed  to  ameliorate  social  problems  (Rogers,  1986).  Both  OA  sys- 
tems and  social  programs  are  generally  implemented  in  field  studies  without 
benefit  of  formal  experimental  controls  or  control  groups.  Both  are  imple- 
mented by  purposive  action  that  is  tied  to  specific,  frequently  qualitative,  ob- 
jectives. The  effects  of  both  OA  systems  (Kling,  1980)  and  social  programs 
(Salasin,  1973;  Suchman,  1977)  have  been  demonstrated  to  be  far-reaching  and 
often  unanticipated. 

A  significant  body  of  research,  called  "evaluation  research"  has  evolved 
to  assess  the  effects  of  social  experiments  (e.g.  Caro,   1971;  Weiss,   1972a, 
1972b;  Suchman,  1974,  1977;  Patton,  1987).  Rogers  (1986,  p.  218)  notes  the 
similarities  between  OA  systems  and  social  programs  and  suggests  that  evalu- 
ation research  is  a  useful  paradigm  for  considering  the  impact  of  OA  systems. 
This  section  discusses  the  implications  of  viewing  OA  systems  as  social  ex- 
periments, and  explores  findings  from  evaluation  research  relevant  to  OA 
evaluation. 


2 

"  The  term  "social  experiment"  is  from  Suchman  (1974). 


A  Model  of  OA  Implementation  Evaluation 

Evaluations  are  an  important  component  of  the  authority  structure  of 
an  organization  (Scott,  Dornbusch,  Busching,  and  Laing,  1967;  Pfeffer,  1978). 
Figure  1  is  a  model  (adapted  from  Suchman  (1977,  1974))  that  is  useful  in  for- 
mulating the  problem  of  evaluating  an  implemented  OA  system.  The  introduc- 
tion of  OA  into  an  organization  creates  multiple,  measurable  effects  (Yj,  Y£,  ••• 
Ym)  (Hedberg,  Edstrom,  Muller,  and  Wilpert,  1975;  Kling,  1980).  Each  of  these 
effects  may  be  explained  by  an  antecedent  condition  (a^,  ^  —  am),  by  the  in- 
troduction of  the  OA  system  (X),  or  by  an  interaction  between  the  OA  system 
and  an  antecedent  condition  (X  *  an).  No  single  antecedent  condition  is  a  nec- 
essary and  sufficient  explanation  for  any  effect.  Further,  measurable  effects 
(Y)  are  presumed  to  link  to  qualitative  consequences  (b^,  D2» ...  bp),  that  repre- 
sent the  ultimate  objectives  or  outcomes  of  the  system.  Evaluation  under  such 
conditions  requires  multicausal  models,  that  include  consideration  of  the  in- 
terdependence of  antecedent  conditions,  observed  effects,  and  consequences 
(Suchman,  1977). 


Insert  Figure  1  about  here 


To  illustrate  the  model,  consider  the  evaluation  of  a  teleconferencing 
system  (X)  designed  to  facilitate  meetings  between  geographically-disbursed 
engineers  or  executives.  Such  systems  are  currently  in  place  at  Digital  Equip- 
ment, AT&T  (Kiesler,  1986),  and  NASA  (Trevino,  Lengel,  and  Daft,  1987).  Po- 
tentially important  antecedent  conditions  (ams)  include  existing  communica- 
tion modes,  power  relationships  between  engineers  and  executives,  manage- 
rial roles,  familiarity  with  communications  technology,  and  the  stated  goals  of 
the  teleconferencing  system.  Measurable  effects  (Yns)  might  include  measures 


8 

of  productivity  (e.g.  time  required  to  make  a  decision),  process  (e.g.  frequency 
of  communications  between  users),  and  OA  product  (e.g.  development  costs, 
on-going  maintenance  costs)  (Sprague  and  Carlson,  1982).  Consequences  of  the 
system  (bps)  might  include  a  better  understanding  of  the  business,  new  in- 
sights, and  improved  communication. 

Inferences  of  causality  under  such  conditions  must  be  conditional  and 
probabilistic,  focused  upon  identifying  contributing  or  precipitating  factors, 
rather  than  determining  conditions.  Because  of  the  complexity  and  equivocal- 
ity of  the  organizational  environment  (Pondy  and  Mitroff,  1978),  effects  re- 
sulting from  OA  implementation  must  be  considered  dependent  upon  existing 
preconditions  present  in  the  environment,  and  upon  the  linkage  of  precondi- 
tions with  system  objectives  and  implemented  technology.  As  Kling  (1980) 
notes,  computers  have  no  intrinsic  impact.  Rather,  the  impact  of  computing  on 
organizations  results  from  the  joint  effects  of  computing  technology  and  man- 
agerial intent: 

The  "consequences"  of  computer  use  are  simply  the  consequences 
of  purposive  action  married  to  computing.  As  these  actions 
change,  the  "consequences"  may  well  change.  ...  "Impact"  [from 
computing]  happens  because  of  some  underlying  patterned  social 
process,  and  speaking  about  the  "impacts  of  technology"  often  dis- 
tracts attention  from  the  social  processes  by  which  they  are 
developed,  adopted,  and  used.  (Kling,  1980,  p.  100). 

OA  systems  can  be  considered  only  one  factor  introduced  into  an  on- 
going social  process  existing  prior  to  implementation.  Suchman  argues  for  a 
similar  perspective  is  assessing  the  effects  of  social  programs: 

"Thus,  any  explanation  of  the  success  or  failure  of  program  [X]  to 
achieve  effect  [Y]  must  take  into  account  the  preconditions  under 
which  the  [system]  is  initiated,  the  events  which  intervene  be- 
tween the  time  the  program  begins  and  the  time  the  effects  are 
produced,  and  the  consequences  that  follow  upon  the  effects. 
Thus  no  program  is  an  entity  unto  itself,  but  must  be  viewed  as 
part  of  an  ongoing  social  system  (Suchman,  1977,  p.  51). 


Measurable  and  Qualitative  Failure 

Two  sources  of  potential  OA  system  failure  can  be  identified  using  Fig- 
ure 1.  Measurable  failure  is  herein  defined  as  failure  of  the  system  (X)  to  im- 
pact the  measurable  dependent  variables  (Yns).  Qualitative  failure  is  herein 
defined  as  the  failure  of  the  measurable  dependent  variables  (Yns)  to  impact 
system  objectives  (bp). 

Measurable  failure  in  OA  systems  can  be  detected  using  low  variety 
evaluation  methodologies.  Once  identified  however,  measurable  failure  is  not 
easily  explained,  due  to  the  quasi-experimental  design  used  in  OA  implemen- 
tation. Possible  explanations  include:  1)  poor  system  engineering  (i.e.  inade- 
quate X),  2)  preexisting  conditions  that  nullified  the  impact  of  the  system  (a  * 
X  interaction),  or,  3)  inadequate  measurement  of  dependent  variables  (i.e., 
Ys).  Techniques  intended  to  improve  low  variety  evaluation  have  been  pre- 
sented in  the  systems  literature  in  discussions  of  experimental  design  (e.g. 
Jarvenpaa,  Dickson,  and  DeSanctis,  1985),  systems  engineering  (e.g.  Sprague 
and  Carlson,  1982),  and  measurement  methodologies  (e.g.  Miller  and  Doyle, 
1987).  Qualitative  failure  has  received  considerably  less  attention  in  the  IS  lit- 
erature (a  notable  exception  is  Mason  and  Swanson,  1981).  Consequently,  the 
remainder  of  this  article  will  focus  on  evaluative  techniques  for  detecting  and 
describing  qualitative  failure. 

Qualitative  failures  are  summarized  metaphorically  by  Suchman  as 
medical  operations  in  which  "the  operation  was  a  success,  but  the  patient  died 
(Suchman,  1977,  p.  51)."  Qualitative  failures  are  attributable  to  a  misspecified 
or  nonexistent  relationship  between  measurable  dependent  variables  and  sys- 
tem objectives.  In  such  cases,  the  system  may  impact  the  dependent  variables 
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as  predicted  (i.e.  X  -->  Yns).  However,  changes  in  the  dependent  variables 
(Yns)  do  not  lead  to  the  expected  consequences  ((i.e.  Yns  -/->  bps)). 

For  example,  an  OA  system  might  reduce  the  time  required  to  make 
decisions,  increase  the  frequency  of  communications  between  executives,  be 
praised  by  end-users,  and  still  be  perceived  as  decreasing  the  quality  of 
organizational  communication.  Reduced  time  to  make  decisions  may  decrease 
decision  quality  by  reducing   time   needed  for  deliberation  and  cogitation 
(Zuboff,  1982;  Weick,  1985).  Increased  quantity  of  communication  between 
users  may  be  due  to  decreased  quality  of  interactions  (e.g.  "I  was  just  calling  to 
clarify  what  I  thought  you  said  during  our  last  conference  call  ...")  (Kiesler, 
1986;  Daft,  Trevino,  and  Lengel,  1987).  End-users  might  accept  the  system  to 
placate  superiors  with  a  vested  interest  in  making  the  system  appear  success- 
ful (Eveland,  1983).  The  above  scenario  provides  an  OA  equivalent  of  Such- 
man's  medical  metaphor:  "No  one  likes  the  system,  but  everyone  uses  it." 

Low  variety  evaluation  methodologies  are  well-suited  to  capturing  mea- 
surable failure,  but  not  qualitative  failure.  Low  variety  measures  of  time  to 
make  a  decision,  frequency  of  electronic  communications,  and  user  perception 
would  likely  indicate  in  the  above  example  that  the  system  was  a  success. 
However,  high  variety  methodologies  would  be  required  to  determine  if  the 
qualitative  consequences  of  the  system  were  desirable  (Rice  and  Rogers  1984; 
Rogers,  1986). 

RATIONAL  DECISIONS  TO  NOT  EVALUATE 

The  potential  for  qualitative  failure  provides  a  justification  for  omitting 
low  variety  evaluative  approaches  of  OA  in  certain  circumstances.  Since  low 
variety  methodologies  are  hypothesized  to  provide  data  relevant  only  to  as- 
sessing measurable  failure,  then  researchers  interested  in  determining  quali- 
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tative  success  or  failure  are  well  advised  to  forego  low  variety  evaluations. 
Further,  evidence  from  existing  OA  systems  suggests  they  have  provided  no 
direct  cost  savings  (National  Archives  and  Records  Service,  1981;  Uttal,  1982; 
Bowen,  1986;  Business  Week,  1988),  yet  have  had  a  significant  impact  on 
qualitative  processes  in  organizations  (Zuboff,  1982;  Kiesler,  1986;  Sproull  and 
Kiesler,  1986;  Jarvenpaa,  1988).  This  suggests  that  OA  systems  may  be  prone 
to  weak  or  nonexistent  causal  links  between  measured  dependent  variables 
and  systems  objectives  (Rice  and  Rogers,  1984;  Rogers,  1986).  As  a  result, 
thoughtful  consideration  of  the  strength  of  linkages  between  measurable  de- 
pendent variables  and  systems  objectives  is  relevant  to  designing  effective  OA 
evaluation. 

The  Measurement  of  Intangible  Benefits 

The  extent  to  which  intangible  benefits  can  be  captured  in  measurable 
variables  is  an  issue  in  the  IS  literature.  Some  authors  argue  that  aH  IS  bene- 
fits can  be  assigned  quantitative  values  (Emery,  1982;  Wallace,  1984).  Others 
contend  that  attempting  to  quantify  intangible  benefits  leads  to  a  misplaced 
focus  on  the  assumptions  used  for  quantification,  and  thereby  distracts  from 
meaningful  evaluation  (Knutsen  and  Nolan,  1974;  Keen,  1975;  Keen,  1981).  In 
a  discussion  of  quantifying  the  benefits  resulting  from  building  controls  into 
accounting  IS,  an  Arthur  Young  and  Co.  training  manual  argues  that  quanti- 
fying intangible  benefits  is  frequently  impossible: 

Determining  the  benefits  ...   to  be  derived  from  correcting  a 
weakness  in  controls  is  even  more  subjective  than  determining 
the  costs.  In  many  (if  not  most)  cases,  the  benefits  to  be  derived 
from  a  given  control  cannot  be  measured  quantitatively  (Arthur 
Young  and  Co.,  1980,  p.  61). 

The  principle  of  incompatibility  provides  an  explanation  for  the  inability 
to  capture  qualitative  phenomena  using  quantitative  analysis  (Daft  and  Wigin- 
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ton,  1979).  By  definition,  intangible  goals  contain  ambiguity  and  interde- 
pendence between  decision  factors.  The  principle  of  incompatibility  suggests 
that  nontrivial  descriptions  of  intangible  goals  requires  the  use  of  language 
that  captures  these  ambiguities  and  interdependencies.  Attempting  to  mea- 
surable intangible,  qualitative  factors  using  low  variety  evaluative  methodol- 
ogy may  result  in  a  caricature,  rather  than  a  veracious  description,  of  such  fac- 
tors. Only  high  variety  evaluation  methodologies  contain  sufficient  complexity 
to  provide  meaningful  evaluations  of  intangible  benefits. 

If  system  objectives  cannot  be  adequately  captured  by  measurable  de- 
pendent variables,  then  undertaking  low  variety  evaluations  of  OA  system  in- 
curs costs  with  no  corresponding  instrumental  benefits.  Existing  evidence  sug- 
gests that  IS  practitioners  often  do  not  employ  low  variety  evaluation  strate- 
gies when  systems  are  developed  in  response  to  intangible  goals  (Keen,  1981; 
Sprague  and  Carlson,  1982;  Hogue  and  Watson,  1984).  In  such  cases,  under- 
taking formal,  quantitative  evaluation  may  be  undertaken  for  primarily  sym- 
bolic rather  than  instrumental  value  (March,  1978;  Feldman  and  March,  1981; 
Meyer,  1984).  For  example,  a  formal  cost-benefit  analysis  of  an  OA  system  may 
be  undertaken  primarily  as  a  ritual  necessary  to  projecting  an  appearance  of 
rational  decision  making  (Robey  and  Markus,  1984). 

The  Goal  Trap 

The  preceding  discussion  suggests  that  under  certain  circumstances, 
low  variety  evaluation  may  result  in  incurring  costs  with  only  symbolic  bene- 
fits. However,  social  intervention  research  indicates  that  improperly-applied 
low  variety  evaluation  may  also  result  in  a  redefinition  of  program  objectives. 
Deutscher  (1977)  defines  the  transformation  of  program  goals  resulting  from 
low  variety  measurement  as  the  "goal  trap." 
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The  goal  trap  refers  to  the  potential  for  the  measurement  process  to 
lead  to  trivialized  programs,  directed  not  towards  original  goals,  but  rather 
towards  achieving  the  measurable  "equivalents"  of  those  goals.  In  terms  of 
Suchman's  model,  the  dependent  variables  (Y)  may  replace  and  usurp  system 
objectives  (bps).  Original,  qualitative  goals  are  replaced  with  the  measurable 
"equivalents".  However,  the  measurable  "equivalents"  may  be  poor  surrogates 
for  the  original  qualitative  objectives  of  the  system. 

The  goal  trap  has  been  recognized  in  IS.  Moore  and  Chang  (1983)  de- 
scribe cases  of  premature  closure  on  systems  requirements  leading  to  creating 
systems  designed  to  address  structured,  easily  solved  problems,  rather  than 
more  important,  but  less  structured  problems.  Similarly,  Kleijnen  observes 
that  emphasizing  quantitative  measures  of  benefits  can  result  in  clerical  appli- 
cations being  given  priority  over  less  easily  justified  projects.  As  a  result: 

"in  more  sophisticated  applications  there  is  the  danger  of  concen- 
trating on  the  easily  quantified  factors  (say,  inventory  cost)  at  the 
expense  of  possibly  more  important,  but  more  difficult  to  quan- 
tify factors  (say,  goodwill)  (Kleijnen,  1980,  p.  9)." 

Colton  (1978)  provides  an  example  of  a  system  that  was  successful  ac- 
cording to  pre-established  measurable  criteria,  but  that  could  be  described  as  a 
qualitative  failure.  The  Kansas  City  Police  Department  provided  officers  with 
decision  support  in  locating  stolen  cars,  unpaid  parking  tickets,  and  unregis- 
tered vehicles.  Police  Department  statistics  showed  a  marked  increase  in  effi- 
ciency as  a  result.  The  (ex)Chief  ultimately  regretted  the  DSS  however,  since 
officers  spent  more  time  on  these  relatively  minor  offenses,  to  the  exclusion  of 
messier,  less  easily  quantified  cases.  Similar  dysfunctional  consequences  from 
the  use  of  low  variety  measures  in  equivocal,  complex  environments  have  been 
observed  in  research  on  welfare  offices  and  public  employment  agencies  (Blau, 
1955),  undergraduate  education  (Becker,  Geer,  and  Hughes,  1968),  military 
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strategy  (Campbell,  1977),  and  both  Soviet  (Granick,  1954)  and  U.S.  (Argyris, 
1952)  factory  production  management. 

STRATEGIES  FOR  DESIGNING  OA  EVALUATION 

The  exclusive  use  of  low  variety  evaluation  methodologies  in  environ- 
ments characterized  by  high  equivocality  creates  the  potential  to  ensnare  both 
evaluator  and  evaluatee  in  the  goal  trap.  Such  arguments  help  explain  why  IS 
practitioners  frequently  omit  low  variety  evaluations  of  IS.  However,  these  ob- 
servations are  not  helpful  irr  designing  techniques  more  appropriate  to  OA 
evaluation.  Accordingly,  this  section  offers  suggestions  for  increasing  the  rele- 
vance of  OA  evaluation,  and  decreasing  its  potentially  destructive  by-products. 

1.  Use  Multiple  Perspectives  For  Evaluation 

One  approach  to  increasing  the  relevance  of  OA  evaluation  is  to  use 
multiple  perspectives  in  evaluation  (Akoka,  1981;  Robey  and  Marcus  1984; 
Cooper  1988).  Kling  (1980)  argues  that  there  are  two  basic  theories  used  to 
examine  the  effects  of  computing.  Weick  summarizes  these  theories  as  fol- 
lows:3 

"Systems  rationalist  perspectives  assume  consensus  on  goals  rel- 
evant to  computer  use,  view  efficiency  as  a  predominant  value, 
focus  attention  on  the  user  more  than  on  the  context  in  which 
the  user  operates,  assume  a  top-down  view  of  change  and  im- 
plementation, and  treat  the  formal  authority  structure  as  an  ac- 
curate map  of  the  way  activities  are  carried  out. 

Segmented  institutionalist  perspectives  assume  that  conflict  is 
more  common  than  consensus,  that  definitions  of  the  situation 
are  multiple,  that  goals  are  diverse,  that  implementation  is  af- 
fected by  vested  interest  groups  and  power,  that  relevant  social 
forms  consist  of  much  more  than  task  groups,  and  that  technol- 
ogy can  take  on  a  variety  of  meanings  (Weick,  1984,  p.  118)." 


-I 
Other  authors  have  presented  similar  dichotomies  using  different  labels.  For  example,  Boland 

(1978)  discusses  the  "traditional"  and  "alternative"  perspectives  of  rationality  in  IS  design.  These 

roughly  correspond  to  Kling's  systems  rationalist  and  segmented  institutionalist  perspectives, 

respectively. 
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Kling's  perspectives  provide  alternative  frames  for  conducting  OA  eval- 
uation. In  terms  of  Suchman's  model,  multiple  evaluative  perspectives  can  be 
used  to  increase  both  the  number  of  orthogonal  dependent  variables  (Ys),  and 
the  number  of  observed  outcomes  (bps).  Increasing  the  pool  of  unique  vari- 
ables and  observed  outcomes  makes  it  less  likely  that:  (1)  only  predicted  con- 
sequences will  be  observed,  (2)  data  will  be  an  artifact  of  the  theoretical  per- 
spective embraced  by  the  evaluator  (Weick,  1979),  and  (3)  evaluation  will  use 
only  low  variety  measures.  Consequently,  a  pragmatic  strategy  for  improving 
OA  evaluation  is  to  integrate  the  systems  rationalist  and  segmented  institu- 
tionalist  perspectives  in  evaluation.  Suggestions  for  accomplishing  this  inte- 
gration follow. 

2.  Attend  to  the  Unintended 

Suchman's  model  of  evaluation  (Figure  1)  illustrates  the  complex,  in- 
terdependent environment  in  which  OA  systems  are  placed.  This  environment 
insures  that  multiple  effects  will  result  from  OA  implementation.  Some  of 
these  effects  will  be  intended;  however,  many  more  will  be  unintended,  and 
unanticipated  by  system  designers  (Rogers,  1983,  1984,  1986).  Recent  research 
confirms  that  unintended  effects  result  from  OA  systems.  Demonstrated  unin- 
tended effects  include  changes  in  leadership  patterns  (Strickland,  Guild,  Bare- 
foot, and  Paterson,  1978;  Zigurs,  Poole,  and  DeSanctis,  1987),  altered  propen- 
sities toward  risk  (McGuire,  Kiesler  and  Siegel,  1987),  interruptions  of  tradi- 
tional feedback  processes  (Zuboff,   1982),  changed  communication  patterns 
(Williams,   1978;   Rogers,    1986),   altered  social   relationships   (Zuboff,    1982; 
Kiesler,  1986;  Siegel,  Dubrovsky,  Kiesler,  and  McGuire,  1986),  changes  in  so- 
cial status  (Rogers,  1986),  increases  in  the  quantity  of  available  information 
(Zuboff,   1982),  and  decreases  in  the  number  and  richness  of  information 
sources  (Zuboff,  1982;  Weick,  1985;  Daft,  Lengel,  and  Trevino,  1987). 
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Kiesler  (1986)  and  Rogers  (1983,  1986)  observe  a  bias  in  the  evaluation 
of  new  technology  towards  overestimating  the  expected  improvements  in  effi- 
ciency, and  underestimating  the  unintended  social  effects.  To  illustrate,  the 
elevator  was  introduced  to  increase  the  efficiency  of  movement  in  multi-floor 
buildings  (Kiesler,  1986).  However,  the  elevator  enabled  skyscrapers  and  ex- 
tremely high  population  densities.  Consequently,  an  unintended  social  effect 
of  elevators  is  increased  superficiality  in  social  contacts,  as  people  work  in  one 
area  and  live  in  another.  A  byproduct  of  this  superficiality  is  greater  alienation 
and  increased  social  distance.  Such  effects  from  elevators  were  unpredicted 
and  unexpected.  Given  the  significant  capacity  of  OA  to  alter  work  relation- 
ships (Zuboff,  1982;  Weick,  1985),  long-term  social  effects  are  likely  to  result 
from  OA  systems. 

Actively  seeking  out  the  unintended  effects  of  OA  avoids  framing  the 
evaluation  question  as  a  nonsensical  null  hypothesis  that  ignores  unintended 
effects.  When  evaluating  OA  systems  implemented  in  complex  social  systems, 
asking,  "did  anything  happen  from  the  OA  implementation?"  is  inane.  The 
interdependencies  present  in  any  meaningful  OA  implementation  insure  that 
the  null  hypothesis  of  "nothing"  happening  is  logically  impossible.  The  more 
sensible  questions  to  be  answered  in  evaluating  OA  systems  are,  "what  is  hap- 
pening that  was  intended?"  and  "what  is  happening  that  was  not  intended?" 
(Merton,  1957;  Deutscher,  1977). 

Reframing  the  evaluation  question  as  stated  above  serves  two  objec- 
tives. Changing  the  temporal  form  of  the  question  (i.e.  from  "what  happened" 
to,  "what  is  happening?")  reframes  the  evaluation  of  OA  systems  from  a  static 
evaluation  of  a  past  event,  into  recognition  of  the  on-going,  interdependent  na- 
ture of  the  social  process  in  which  even  the  evaluator  is  a  party  (Pondy,  1977). 
Such  a  perspective  aids  the  evaluator  in  recognizing  that  the  evaluation  pro- 
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cess  (as  well  as  the  evaluation  product)  is  likely  to  impact  perceptions  of  the 
OA  system,  and  may  therefore  change  perceptions  of  the  success  or  failure  of 
the  system.  Dividing  the  evaluation  question  into  consideration  of  both  the  in- 
tended and  unintended  effects  of  the  system  further  insures  that  OA  evalua- 
tion does  not  exclusively  focus  on  predefined  variables  that  the  system  was 
predicted  to  impact.  The  complexity  of  the  social  environment  of  an  OA 
implementation  makes  prediction  of  all  effects  impossible.  However,  the  un- 
predictability of  OA  system  effects  does  not  mean  that  such  effects  cannot  be 
recognized  and  evaluated  ex  post. 

It  is  a  logical  contradiction  to  suggest  that  the  unanticipated  effects  of 
OA  can  be  predicted.  However,  explicating  and  evaluating  unintended  effects 
should  be  an  essential  component  of  OA  evaluation.  Possible  unintended  ef- 
fects of  OA  can  be  obtained  from  a  variety  of  sources.  Existing  research  in  OA 
explicates  many  of  the  short  term  unintended  consequences  of  OA  (Williams, 
1978;  Zuboff,  1982;  Kiesler,  1986;  Er,  1987).  Extrapolating  from  the  long-term 
effects  of  prior  technological  advances,  such  as  the  printing  press,  elevator, 
and  telephone  offer  a  second  source  (Kiesler,  1986;  Rogers,  1986).  Finally,  op- 
ponents of  OA  implementation,  both  within  and  outside  the  implementing 
organization  can  identify  many  potential  undesirable  consequences  of  OA 
(Salasin,  1973;  Rogers,  1984). 

3.  Triangulate  Data  Sources 

One  strategy  for  increasing  the  likelihood  of  capturing  unintended  ef- 
fects is  the  use  of  multiple,  independent  data  sources.  Many  authors  have  sug- 
gested the  use  of  multiple  measures  as  a  means  for  avoiding  the  goal  trap  (e.g. 
Ridgway,  1981).  However,  the  use  of  multiple  measures  is  a  necessary  but 
insufficient  condition  for  triangulation  in  measurement   (Webb,  Campbell, 
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Schwartz,  and  Sechrest,  1966;  Isaac  and  Michael,  1984).  The  use  of  multiple, 
low  variety  measures  (e.g.  a  series  of  survey  questions),  or  multiple  measures 
within  a  single  perspective  (e.g.  multiple  measures  of  user  satisfaction)  does 
little  to  increase  evaluative  variety.  For  example,  existing  data  suggests  that 
low  variety  measures  of  user  satisfaction  are  largely  redundant  (Rushinek  and 
Rushinek,  1983).     • 

Table  1  provides  a  reference  source  for  generating  triangulated  data.  By 
choosing  one  evaluative  approach  from  the  upper  half  of  the  list,  and  one  from 
the  lower  half,  evaluators  can  avoid  the  false  security  of  having  multiple,  con- 
firming low  variety  measures.  For  example,  one  evaluative  strategy  for  OA 
might  be  to  "go  native"  and  conduct  covert  field  research,  but  combine  this  ap- 
proach with  a  user  questionnaire.  Such  an  approach  would  likely  provide  in- 
sights into  social  relationships  and  unintended  consequences  from  OA  that 
would  be  overlooked  by  exclusive  use  of  a  questionnaire. 

4.  Evaluate  Processes  And  Input/Output 

A  second  method  for  triangulating  data  is  to  combine  input/output 
analysis  with  evaluations  of  social  (Rogers,  1986)  and  decision  processes  (Todd 
and  Benbasat,  1987).  OA  systems  are  increasingly  designed  to  impact  man- 
agerial decision  making  processes  (McLeod  and  Jones,  1987).  Consequently,  an 
evaluative  perspective  that  focuses  solely  on  input  and  outputs  (e.g.  quantifi- 
able costs  and  benefits)  is  likely  to  fail  to  capture  much  of  the  impact  of  OA. 
Integrating  input-output  measures  with  evaluations  of  processes  offers  a 
richer,  more  complete  evaluative  perspective  than  can  be  provided  by  input- 
output  analyses  alone  (Hedberg,  Nystrom,  and  Starbuck,  1976;  Payne,  Braun- 
stein,  and  Carroll,  1978;  Einhorn,  Kleinmuntz,  and  Kleinmuntz,  1979;  Rogers, 
1984;  Patton,  1987)). 
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5.  Discover  Goals,  Don 't  Evaluate  Them 

Recent  research  suggests  that,  "organizations  are  rarely  what  they  pre- 
tend to  be  (Deutscher,  1977,  p.  221)."  Significant  differences  often  exist  in  the 
formal  and  informal  structures,  processes,  and  goals  of  organizations  (March 
and  Olsen,  1976;  Weick,  1976;  Sproull,  Weiner,  and  Wolf,  1978).  Managers  may 
only  understand  and  make  sense  of  events  after  the  events  have  transpired 
(Hirschman,  1967;  Weick,  1979).  Goals  may  be  left  intentionally  vague,  or  even 
stated  as  opposites  of  desired  ends  in  order  to  strategically  manage  prefer- 
ences for  outcomes  (March,  1978;  Pascale,  1979).  Within  IS,  rituals  may  be 
used  to  "...  symbolize  rationality  and  signify  that  the  actions  taken  are  not 
arbitrary,  but  rather  acceptable  within  the  organization's  ideology"  (Robey  and 
Markus,  1984,  p.  12).  Such  examples  suggest  that  OA  systems  may  be  con- 
structed for  reasons  other  than  those  stated  in  a  formal  information  require- 
ments analysis.  Consequently,  the  formally  stated  goals  of  an  OA  system  may 
have  little  to  do  with  the  actual  goals  of  managers,  designers,  and  users. 

The  loose  coupling  of  formal  and  actual  goals  in  organizations  suggests 
that  evaluation  directed  only  at  measuring  the  extent  to  which  stated  goals 
have  been  met  by  an  OA  system  will  likely  miss  most  of  what  goes  on  in 
organizations  (Deutscher,  1977;  Deutscher  and  Gold,  1979).  Actual  goals  and 
unintended  consequences  are  intimately  wound  into  the  on-going  social  mosaic 
within  which  OA  is  placed.  As  a  result,  a  major  objective  of  OA  evaluation 
should  be  to  explicate  the  goals  of  organization  actors  responsible  for  an  OA 
system. 

Evaluating  OA  systems  from  this  perspective  replaces  the  task  of  evalu- 
ating the  extent  to  which  the  system  has  met  its  original  goals,  with  the  task 
of  discovering  the  system's  actual  goals.  Such  an  approach  assumes  that  users, 
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designers,  and  managers  often  cannot  or  will  not  articulate  the  actual  goals  of 
OA.  In  such  cases,  evaluation  becomes  similar  to  investigative  journalism 
(when  actors  will  not  state  goals)  (Guba,  1981)  or  participant-observer  re- 
search (when  actors  cannot  state  goals)  (Deutscher  and  Gold,  1979).  Either 
role  requires  an  evaluative  perspective  relatively  unexplored  in  IS. 

CONCLUSION 

The  frustration  of  IS  professionals  with  existing  methodologies  for  OA 
evaluation  is  expressed  by  a  Westinghouse  Electric  executive  responsible  for 
monitoring  the  company's  use  of  teleconferencing: 

"I  can  measure  how  much  this  video  teleconferencing  equipment 
costs  and  guess  about  how  much  we  save  in  travel  expenses  when 
we  apply  it,  but  I  haven't  the  slightest  idea  how  good  the  decision 
are  that  our  people  reach  when  they  use  it.  (Kiesler,  1986,  p.  54)" 

The  principle  of  incompatibility  suggests  that  accurate  and  consequen- 
tial statements  about  complex  organizational  communication  systems  are  not 
possible  with  low  variety,  single  perspective  evaluations.  Such  methodologies 
do  not  contain  sufficient  complexity  to  permit  description  and  evaluation  of 
equivocality  management  and  organizational  sense-making  processes  (Pondy, 
1977;  Daft  &  Wiginton,  1979).  Rather,  effective  evaluation  in  equivocal  envi- 
ronments   requires    integrating    multiple    perspectives,    observing    the    un- 
expected, collecting  orthogonal  measures,  measuring  processes,  discovering 
goals,  and  employing  both  high  and  low  variety  methodology. 

This  paper  began  with  the  observation  that,  in  evaluating  the  effects  of 
IS,  practitioners  rarely  practice  what  researchers  preach,  and  researchers 
rarely  preach  what  practitioners  practice.  Ultimately,  the  point  of  this  paper  is 
that  both  IS  evaluation  research  and  practice  can  be  improved  by  integrating 
the  seemingly  disparate  perspectives  of  practitioners  and  researchers.  IS  re- 
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search  can  be  improved  by  greater  attention  to  and  observation  of  existing 
practice,  unencumbered  by  existing  normative  models  of  evaluation.  IS  prac- 
tice can  be  improved  by  experimentation  with  formal  models  and  theories  of 
evaluation,  leading  to  improvements  in  current  practice. 

The  implications  of  this  argument  for  IS  research  are  more  subtle  and 
profound  than  may  be  initially  evident.  The  choice  of  a  methodology  for  evalu- 
ation of  OA  is  tightly  bound  to  issues  of  theoretical  substance.  That  is,  what 
evaluators  "see"  when  assessing  the  effects  of  technology  is  largely  a  function 
of  what  they  expect  to  see  (i.e.  "believing  is  seeing,"  Weick,  1979,  p.  135).  Re- 
searchers using  low  variety,  single  perspective  evaluative  methodology  expect 
to  "see"  rational,  measurable  phenomena  that  are  largely  predictable  and  or- 
derly. However,  unintended  effects,  social  processes,  and  goal  displacement 
and  evolution  generally  lie  outside  of  the  rational  models  and  measurements 
often  prescribed  for  OA  evaluation.  As  Weick  suggests: 

"The  crux  of  my  concern  is  that  the  comfort  MIS  researchers  feel 
with  rational  models,  measurement,  and  order  will  tempt  them 
into  the  following  illogic: 

1.  Every  real  phenomenon  can  be  measured. 

2.  If  it  can't  be  measured,  it's  not  real. 

3.  If  it  can  be  measured,  it's  real. 

Blunt  measures,  narrow  measures,  and  nonmeasures  poke  holes 
in  this  reasoning.  Nevertheless,  the  imagery  is  seductive.  Appli- 
cation of  this  illogic,  especially  when  the  issues  in  technology  and 
organization  are  at  a  formative  stage,  could  make  it  impossible 
for  us  to  learn  about  this  important  area  (Weick,  1984,  p.  129)." 

The  insights  of  social  intervention  research  offer  an  alternative  to  the 
current  schism  between  IS  evaluation  research  and  IS  evaluation  practice. 
Further,  such  methodologies  are  not  new  and  untested,  but  rather  are  the 
cumulative  result  of  30  years  of  research  into  social  interventions  and  organi- 
zational processes.  An  old  cliche  goes,  "you  can't  teach  an  old  dog  new  tricks." 
The  important  question  for  OA  researchers  and  evaluators  in  their  decision  to 
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apply  and  adapt  the  insights  of  social  intervention  research  is  however,  "Can 
you  teach  old  tricks  to  a  new  dog?". 
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18.  Judicial  Investigations 

(operating  under  rules  of 
evidence) 

19.  Panel  (test  and  retest)  Studies 

20.  Laboratory  Experiments 

21.  Questionnaires  and  Polls 

22.  Computer  Simulation  Studies 

23.  Mathematical  and  Statistical 

Models 
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